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Summary 

The  rapid  development  of  graphics  technology  allows  for 
greater  flexibility  in  aircraR  displays,  but  display  evalua¬ 
tion  techniques  have  not  kept  pace.  Historically,  display 
evaluation  has  been  based  on  subjcctisc  opinion  and  not 
on  the  actual  aircraft/pilot  performance.  Existing  elec¬ 
tronic  display  specifications  and  evaluation  techniques  arc 
reviewed.  A  display  rating  technique  analogous  to 
handling  qualities  ratings  has  been  developed  and  K 
recommended  for  future  evaluations.  The  choice  of 
evaluation  pilots  is  also  dtscus.scd  and  the  use  of  a  limited 
number  of  trained  evaluators  is  recommended  over  the  use 
of  a  larger  number  of  operational  pilots. 

1.  Introduction 

The  head-up  display  (HUD)  is  becoming  th '  primary 
fixed-wing  flight  reference  for  use  during  both  v-sual  and 
instrumental  meteorological  conditions.  An  otfspring  of 
the  HUD  technology,  the  helmet-mounted  disp'ay 
(HMD),  has  been  developed  to  accommodatr  the  require¬ 
ment  for  larger  flcld-of-rcgard  displays.  The  HMD  is 
expected  to  become  a  primary  rotary-wing  flight  reference 
in  the  future. 

HUD  and  HMD  allow  the  presentation  of  flight-cniical 
information  in  a  plethora  of  formats.  This  technology 
influx  creates  the  potential  for  new  and  unique  formats  for 
information  critical  to  flight  and  mission  success  to  be 
conveyed  to  the  flight  crew.  The  historical  methods  of 
testing  flight  displays  must  be  improved  and  updated  tc 
provide  vcnflablc  objective  evaluations  of  HUO  and 
HMD. 

This  document  addresses  the  issue  of  evaluating  the  HUD 
or  HMD  symbology  formats  for  use  a.s  primary  flight 
references,  although  these  observations  apply  to  other 
flight  displays. 

1.1  A  Brief  History  of  HDDs  tnd  HMDs 

The  HUD  is  an  outgrowth  of  World  War  li  reflecting 
gunsights.  Gunsights,  which  began  as  simple  iron  rings, 
developed  into  collimated  displays  reflected  from  a 


semitransparent  combiner  glass.  The  benefit  of  collimated 
virtual  image  for  the  pilot  was  the  ability  to  fiKUs  on  both 
the  target  and  the  sight,  rather  than  having  one  appear 
blurred  or  doubled  The  result  of  this  development  was  the 
lead-compensating  optical  sight.  C.s.scntial  flight  informa¬ 
tion  in  the  HUD — such  as  airspeed  or  altitude — was  also 
included  to  aid  the  pilot  in  maintaining  an  eyes-out 
orientation.  As  airborne  computer  graphics  technology 
advanced  over  the  next  decades,  the  HUD  evolved  to  a 
miniature  instrument  display 

Hie  major  advantages  of  HUD  and  HMD  arc  seemingly 
obvious* 

Reduced  pilot  workload-  Pilot  workload  is  reduced 
when  the  overall  pOoting  tasks  require  head-up.  outside- 
ihc-cockpit  flight  references. 

Inerrused  flight  p.^'ccLsion-  The  overlay  of  HUD/HMD- 
presented  flight  data  on  the  external  visual  scene  allows 
the  pilot  to  fly  more  precisely 

Direct  visualization  of  trajectory-  A  conformal  display 
allows  the  pitot  to  directly  assevs  the  aircraft  performance. 

lncrfa.sed  flight  safety-  tvscniial  flight  information 
presented  on  the  HUD/HMD  reduces  eyes-m  thc-cockpit 
during  critical  flight  maneuvers. 

In  the  early  |QK(X.  an  HMD  was  developed  for  the  U.S. 
Army\s  AH-64  Apache  attack  hcliceiplcr.  The  AH-64 
HMD.  when  integrated  with  the  Pilot  Night  Vision  Sensor 
system  (PNVS).  provided  night  vision  information  for 
pilotage  and  weapons  aiming  during  nap-of-thc-carth 
flight  (ref.  I)  Night  vision  video  imagery  from  the  AH-64 
infrared  sensor  is  combined  with  symbology  for  presenta¬ 
tion  at  the  HMD.  While  the  PNVS  .sy.sicm  has  increased 
the  U.S.  Army's  rotorcrafl  mght  and  all-weather  opera¬ 
tions  capability  during  nap-of-thc-carth  operations,  the 
added  dimension  of  off-axis  head  movement  and  sensor 
video  combined  with  symbology  has  added  new 
challenges  for  symbolic  display.s. 

1.2  Display  Formal  Criteria 

Since  the  late  i970s,  a  number  of  reports  have  been 
rclca.scd  citing  significant  deficiencies  in  HUD  symSioiogy 
and  installation.  The  Air  Force  Instrument  Flight  Ccntci 
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(AFIFC)  found  HDDs  were  limited  bj  serious  drawbacks, 
including  lack  of  failure  detection,  lack  of  standardization, 
and  an  increased  tendency  toward  spatial  disorientation 
(ref.  2).  The  HMD  has  only  recently  been  introduced,  so 
analyses  and  studies  of  these  displays  arc  not  readily 
available 

While  there  are  general  specifications  for  military  HUDs 
(ref.  3)  and  HMDs  (ref  4),  the  HUD  symbology  described 
has  not  been  applied  to  any  design.  The  helicopter  HMD 
specifications  agree  with  those  of  the  AH-64 

Traditionally,  electronic  displays  and  the  associated 
symbology  have  been  procured  as  part  of  the  ’•‘.rframe 
weapon  system,  not  as  part  of  “aircraft  instniments  ” 
Classed  as  contractor  furnished  rather  than  government 
furnished  equipment,  adherence  to  general  military 
standards  and  specifications  has  not  been  required  for 
systems  like  the  HUD  Symbology  drive  laws  and 
dynamics  arc  frequently  missing  from  the  spccincaiions 
for  Iwth  HUDs  and  HMDs 

Since  HUDs  were  not  considered  “flight  instruments,’* 
little  need  w  as  seen  to  establish  their  suitability  for  use  as 
a  flight  reference  Consequently,  few  flight  procedures 
were  dcvc'oped  and  limited  training  was  provided  to 
pilots  on  how  to  use  the  HUD  m  routine  flight 

The  only  HMD  flclded  to  date  f<hc  AH-64)  was 
principally  introduced  to  enhance  visual/forward  lookiiig 
Infrared  (FUR)  cues  for  pilotage.  As  a  result,  AH-64 
pilots  arc  trained  to  use  the  HMD  for  flight  purposes 
However,  the  flight  symbology  has  not  been  validated  for 
use  as  a  flight  format  If  a  pilot  enters  mstrument 
meteorological  condiMons  (IMC)  during  low-level  night 
flight,  procedures  dictate  reverting  to  conventional  panel- 
mounted  instruments 

The  reported  deficiencies  in  both  HUD  and  HMD  would 
have  been  corrected  during  flight  tests  had  they  occurred 
in  conventional  panel  instruments.  However,  because  of 
an  absence  of  pcrfomi.incc  based  objcctis  c  criteria,  the 
HUD  display  evaluations  have  relied  on  subjective 
opinion  polls 

1.3  The  Future  Summary  of  Trends  in  Displays 

Today’s  cvKkpit  technology  Is  progressing  almost  fa.stcr 
than  we  can  write  about  it  and  advances  in  clcctmmc 
display  systems  almost  defy  description  It  seems  certain 
that  future  transport  and  tactical  aircraft  will  have  ciKkpits 
with  all-glass  displays  and,  at  nuw,  a  few  conventional 
instruments  for  standby  purposes.  In  addition,  aero 
dynamic  dictates  of  hypersonic  transport  or  combat 
survivability  may  eliminate  direct  external  vision  in  future 
<x»ckpils. 


HMDs  IS  ill  likely  continue  to  progress  from  limited  field 
of  view  (I  OV')  imagery  presented  to  one  eye  to  full  FOV 
HMD  presented  tu  bvith  eyes  with  inprovcd  resolution. 

To  illustrate,  by  the  late  IdQOs  plans  include  progression 
from  30“  x  40“  FOV  monocular  HMD  (in  the  AH-64)  to  a 
30“  X  52“  total  FOV  dual  optic  HMD  (in  the  RAH-66). 

The  basic  question,  however,  remains  Will  we  develop  a 
performance-based  mcthcxiology  *or  evaluation  of  HUDs 
and  HMDs  or  continue  to  rely  on  a  majority  vote  of 
pilots? 

2.  A  Review  of  Display  Symbology 

2.1  Comparison  of  Displays 

Table  I  li<4s  some  characteristics  of  traditional 
in^tfumcnts  and  modern  electronic  displays  The 
conventional  instrument  panel  (round  dials)  is 
characteristically  fixed  in  p('sition  and  has  very  limited 
ability  to  he  progiamnicd  for  dificrcni  flight  segments. 

C  nvcntional  instruments  can  be  color  coded  and  are 
useful  for  display  ir.g  >\  stems  data.  The  pilot  must  look 
tnsJe  the  cockpit  to  observe  the  instruments  since  they  do 
not  appear  in  the  pilot's  view  of  the  real  world. 

Head-down  display.s  (HDDs)  using  cathode  ray  tubes 
(CRTs)  have  many  of  the  same  characteristics  as 
conventional  panels,  but  it  is  possible  to  reprogram  the 
same  display  for  different  phases  of  flight.  For  example, 
an  ctcclronic  attitude  (director)  indicator  can  display 
different  type.s  and  amounts  of  information  during  cruise, 
instrument  approach,  or  t.ikcoff  Tlic  electronic  display 
can  also  generate  symbology  that  is  a  real  world  repre- 
Ncntation,  the  contact  analog  This  has  been  extended  to 
clcctmmc  moving  map  Jispl.iys,  which  arc  analogs  of  the 
world  when  viewed  from  above  Finally,  the  electronic 
CRT  display  can  integrate  data  from  a  number  of  sources, 
including  the  display  of  a  velocity  vector 

HUDs/HMDs  share  some  of  the  characteristics  of  CRT 
displays  These  arc  the  abilities  to  he  programmed,  to  time 
share,  and  to  display  integrated  information  from  a  variety 
of  sources  Although  color  coding  HUDs/HMDs  has  been 
discussed,  it  seems  unlikely  that  cither  will  have  the  same 
dcgrccofcolor  codmgavailabic  in  amvcntiunal  or 
cicctrcmc  hesd-down  instruments  anytime  in  the  near 
future.  Perhaps  the  most  compelling  difference  between 
HUDs/HMDs  and  .ill  other  displays  is  the  ability  of  the 
HUD/HMD  to  display  real  world  conformal  images. 


Table  1.  Display  characteristics 


Display  characteristics 

Round 

dials 

HDDs" 

rvos* 

HUDs 

HMDs 

In  forward  view 

X 

X 

X 

Collimated 

X 

X 

Color  coded 

X 

X 

Programmable 

X 

X 

X 

Time  share 

X 

X 

X 

Integration  possible 

X 

X 

X 

Fovea!  cucs^ 

X 

X 

X 

X 

Peripheral  cues^ 

X 

e 

e 

Useful  for  systems 

X 

X 

f 

r 

Contact  analog  possible 

X 

X 

X 

Conformal  display  possible 

X 

X 

C^n  show  flight  path 

X 

X 

X 

X 

^Head-down  displays  using  CRTs. 

^Peripheral  vision  display.s. 

^Foveal  cues  arc  those  that  require  the  pilot  Hx  his  attention  on  the  display. 
^Peripheral  cues  do  not  require  the  ptlol*s  visual  attention. 

^Questionable  value  with  restricted  FOV. 

/Caulion/warning  displays  only.  Additi<Mial  system  displays  can  add  excessive 
clutter. 


2,2  Publbhed  Specincations 

A  review  of  existing  electronic  display  standards  and 
specifications  shows  a  limited  number  of  standardization 
attempts.  Current  Reifications  and  standards  for 
electronic  HUDs,  HDDs,  or  HMDs  arc  listed  in  (able  2 
Five  of  these  specifications  apply  to  military  aircraft;  four 
to  civil  transport  aircraft;  and  one  applies  to  both  civil  and 
military  aircraft.  Of  the  civil  transport  documents,  (s\o  arc 
industry  recommended  standards,  one  is  an  Advisory 
Circular,  and  one  is  a  draft  Advisory  Circular 

HMD  symbology  standards  arc  largely  an  outgrowth  of 
existing  standards  for  HUDs  with  the  addition  of 
specialized  symbol  and  symbol  driver  requirements  for 
hovering  flight.  To  date,  these  specifications  have  had 
little  impact  on  the  development  of  any  HUD  or  HMD 

There  have  been  several  critical  revievs^  of  HUD 
specifications.  In  the  mid-  to  late  t970s,  the  U.S.  Naval 
Aeromcdicai  Research  Laboratory  reviewed  existing 
HUD  specitications  and  found  a  lack  of  data  to 
substantiate  these  specifications  (refs  I4an<1 15) 

In  the  mid-1970s,  the  Air  Force  Instrument  Flight  Center 
found  that  pilots  had  developed  their  own  techniques  for 


using  the  HUD  and  were,  in  fact,  using  the  HUD  as  a 
flight  reference  (ref.  2).  While  the  HUD  did  represent  a 
significant  aid  as  a  flight  reference,  its  reported  usefulness 
was  limited  by  several  drawbacks:  the  lack  of  adequate 
failure  detection,  inadequate  standardization,  and  a 
reported  increase  in  tendency  toward  spatial  disorienta¬ 
tion.  Foliow-on  studies  have  raised  similar  symbology 
issues. 

In  the  early  1980s,  two  independent  studies  reviewed 
HUD  specifications  (refs  16  and  17).  These  reviews 
found  that  there  was  little  objective  data  (c  substantiate 
specifications,  evaluations,  or  design  choices.  In  the 
absence  of  objective  performance  data,  most  spccifl- 
cations  were  found  to  be  based  on  subjective  opinion. 
Furthcimorc,  utility  as  a  flight  reference  had  not  been 
considered. 

Following  these  studies,  the  U.S.  Air  Force  sponsored  a 
program  to  develop  HUD  criteria.  The  result  was  a  guide 
to  a.s.Mst  the  HUD  designer  to  ensure  that  (he  next 
generation  of  HUDs  would  be  adequate  for  their  (asks 
(ref  7).  While  piovidtng  design  guidance,  an  evaluation 
methodology  was  still  absent. 
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Table!.  Electronic  display  standards 


Specification 

Military 

Civil 

HDD 

HUD 

HMD 

Reference 

MIL-D-81641AS 

X 

X 

3 

MIL-STD-884C 

X 

X 

X 

<2 

5 

MIL-STD-1295 

X 

a 

X 

4 

MIL-STD-1787 

X 

X 

X 

a 

6 

AFWALTR-87.30SS 

X 

X 

X 

7 

AFlFCTR-91-01 

X 

X 

9 

SAE  ARP-4053 

X 

X 

10 

SAEAS-8034 

X 

X 

11 

FAA  AC-25-11 

X 

X 

12 

FAA  draft  paper 

X 

X 

13 

^Not  discussed  in  specification.  However,  the  display  type  shown  is  within  the 
scope  cf  the  specification. 


In  1989,  the  U.S.  Army  Human  Engineering  Laboratory 
started  a  critical  review  of  HMD  requirements  (ref.  8) 
Approximately  100  documents  were  reviewed  and 
performance  data  were  found  to  be  lacking. 

In  the  absence  of  objective  requirements,  all  an  evaluation 
pilot  can  do  is  determine  the  ability  to  fly  by  reference  to 
the  display  without  excessive  workload.  It  is  difficult  to 
document  an  unacceptable  display,  particularly  without 
performance  criteria. 

Tbere  have  been  a  number  of  recurring  problems  with 
HUD  specifications.  The  most  common  are: 

No  dyntnk  requirements-  None  of  the  government 
display  specifications  list  any  dynamic  response  require* 
ments,  other  than  “shall  be  free  from  unacceptable  jitter.** 
The  q>eciflcations  also  fail  to  specify  any  sampling 
interval.  As  system  capabilities  grow,  increased  computer 
workload  can  force  the  computation  interval  to  grow  from 
20-40  msec  to  80-100  msec.  At  some  point  in  the 
lengthening  of  this  interval,  the  display  quality  will 
degrade  dramatically. 

There  appears  to  be  a  misconception  that  100  msec  is  a 
magic  computation  interval,  below  which  there  will  be  no 
display  problems.  This  seems  to  be  based  on  the  idea  of  a 
1/10  second  human  reaction  time.  In  fact,  sampling 
intervals  of  100  msec  can  seriously  degrade  tracking  in 
fighter  aircraft  (ref.  18). 

Standardlzatioii-  HUD  specifications  show  a  complete 
lack  of  standardization.  As  an  example,  in  many  HUOs 
the  angle  of  attack  (AOA)  is  shown  by  an  error  bracket 
that  moves  relative  to  the  velocity  vector.  Some  show  a 
fast  error,  as  the  AOA  bracket  above  the  velocity  vector, 
others  reverse  this. 


The  reason  for  choosing  the  fast  error  abov^slow  error 
below  is  based  on  the  conventional  fly-to  philosophy 
common  in  navigation  deviation  indicators — if  you  arc 
fast,  pull  the  nose  up 

The  background  for  the  reversed  error  sensing  goes  back 
to  the  Klopfstein  format  (ref.  19),  which  made  use  of  the 
relationship  between  AOA,  flight  path  angle,  and  pitch. 
The  f1y»from  AOA  error  bracket  was  intended  to 
emphasize  this  unique  relationship. 

With  rational  arguments  favoring  both  the  fly>to  and  fly* 
from  senses,  which  is  better  m  a  HUD?  At  this  point,  the 
answer  is  not  clear;  however,  it  is  obvious  that  having 
both  arrangements  in  similar  aircraft  has  the  potential  for 
problems.  There  should  be  an  objective  method  of 
determining  the  better  format.  This  method  should  be  a 
performance-based  evaluation. 

Hidden  specifkatloiis-  Finally,  there  are  several 
“hidden**  specifications.  One  example  was  the  100  msec 
computer  frame  time  mentioned  before.  Another  is  the 
{^'ccession  that  occurs  as  the  airplane  passes  ±90*  in  pitch. 
This  IS  a  carryover  from  clcctro-mechanical  attitude 
indicators  to  prevent  gimbal  lock.  An  electronic  display 
has  no  need  to  keep  this  feature.  Yet,  many  HUD 
designers  fee!  that  it  is  an  essential  feature— one  designer 
even  stated  that  there  was  a  military  specification 
requiring  such  a  precession. 

Gold-plated  specincatloiis-  Many  recent  standardization 
attempts  have  been  based  on  a  “wish  list"  for  HUDs  that 
will  60  everything.  In  the  civilian  and  military  communi¬ 
ties,  the  drafters  of  requirements  assume  that  all  future 
aircraft  will  carry  wide  FOV  holographic  HUDs  with  a 
complete  inertial  navigation  system  and  precision  distance 
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measuring  equipment  (DME)  available  The  draft 
spcciHcations  appear  to  preclude  non-conformal  HUDs 
for  many  smaller  corporate  aircraft. 

When  drafting  specifications  and  standards,  there  arc 
places  for  displays  with  narrow  FOVs  driven  by  gyro 
platforms  presenting  air*mass  data.  These  HUDs  may  not 
allow  us  to  fly  to  Category  HI  minimums,  but  they  may 
still  enhance  the  mission  for  which  they  are  intended. 

23  Need  for  Standardization 

The  need  for  absolute  standardization  in  electronic 
displays  is  questioned.  There  appears  to  be  a  strong  desire 
to  have  fighter  HUD  symbologies  the  same  in  alt  aircraft. 
This  IS  surprising  since  there  appears  to  be  little  or  no 
standardization  in  fighter  instrument  panels. 

The  major  reason  for  standardization  is  to  reduce  negative 
habit  transfer  and  allow  pilots  to  move  rapidly  from  one 
airplane  or  system  to  another.  Pilots  today  do  not  jump 
from  one  airplane  to  another  and  reaching  back  to  prior 
training  at  critical  points  can  be  inappropriate. 

In  spite  of  this,  standardization  mu.st  play  a  secondary  role 
to  the  effectiveness  of  the  display  for  the  particular 
aircraft  and  mission.  White  some  aspects  of  standardi* 
zation  should  not  be  changed  arbitrarily  (such  as  airspeed 
on  (he  left  and  altitude  on  the  right  or  the  shape  of  some 
primary  symbols),  variations  in  mission,  aircraft 
performance  and  agility,  sensors  available,  and  HUD 
FOV  should  allow  flexibility  in  symbology  standards.  Wc 
should  be  suipn'scd  if  a  transport  or  a  helicopter  HUD 
were  to  look  like  a  fighter  HUD. 

In  addition,  it  is  more  important  for  modes  within  a  given 
HUD  to  be  consistent  than  to  have  standardization  acrovs 
aircraft.  This  argument  is  based  on  the  pilot  of  a  given 
aircraft  who  changes  aircraft  infrequently  being  exposed 
to  multiple  formats  in  the  same  aircraft  on  a  daily  basis 

For  example,  use  of  a  variable  compression  pitch  scale 
could  have  significant  advantages  during  HUD  instrument 
modes,  but  could  present  difficulties  during  an  air-to- 
ground  (A/G)  weapon  delivery  mode.  In  this  case,  an  A/G 
airplane  should  not  use  variable  compression  pitch  scales 
in  any  mode,  even  if  the  ‘‘standard*’  instrument  mode  uses 
variable  compression. 

Historical  HUD  symbology  problems  were  caused  by 
inappropriate  symbology,  net  by  non-staf>dard 
symbology.  Wc  must  not  become  slaves  to  standardi¬ 
zation  for  its  own  sake.  Historical  symbology  standards 
may  reflect  the  limitations  of  symbol  generators  at  the 
time  they  were  developed  and  should  not  be  allowed  to 
restrict  development  of  advanced  display  formats  The 
primary  goal  should  be  enhanced  pilot/aircraft  perfor¬ 


mance  wiih  HUDi  designed  and  tested  Hiih  mission 
performance  in  mind. 

2.4  Display  Design  Principles 

Traditionally,  display  designers  have  sought  expert  pilot 
opinion  for  guidance  during  the  development  of  new 
flight  displays  While  user  opinion  can  be  helpful,  pilots 
tend  to  have  diverse  (and  strongly  held)  opinions.  In 
addition,  pilots  with  limited  background  in  display 
evaluation  often  limit  the  design  of  novel  sy.stcms  to  those 
concepts  with  which  they  are  familiar. 

The  display  design  must  consider  why  the  pilot  needs  the 
data  and  what  the  pilot  is  expected  to  do  with  the  data. 
According  to  Singleton  (ref  20).  the  following  questions 
should  be  considered  during  the  display  development 

1.  Docs  the  pilot’s  need  justify  the  display? 

2.  Have  all  the  necessary  data  been  provided  to  the 
p**''!?  If  not,  what  additional  data  arc  required? 

3.  Can  the  average  pilot  easily  obtain  the  required  data'’ 

4.  Docs  the  display  conform — 

•  to  the  real  world’ 

•  to  other  cockpit  displays? 

•  with  previous  pilot  habits  and  skills'’ 

•  with  required  decisions  and  actions? 

Following  completion  of  the  display  design,  its  evaluation 
^uld  be  based  on  objective,  performance-based  criteria 
and  measures  of  the  display's  effect  on  mission  perfor¬ 
mance.  it  IS  up  to  the  evaluation  tcim  to  determine  what 
arc  suitable  performance  measures  These  should  reflect 
the  intended  mission  of  the  aircraft  and  should  include  all 
mission  segments 

AH  displays  have  a  need  to  minimize  display  clutter  and 
this  is  particularly  critical  with  see-through  displays. 

Since  HUD/HMD  symbols  arc  presented  in  the  pilot’s 
view  of  the  real  world,  obtrusive  symbology  should  be 
kept  to  an  absolute  minimum  Not  one  “pixel”  should  be 
III  unless  it  “buys”  its  way  onto  the  screen  by  providing  a 
demonstrable  improvement  in  performance  (ref.  21). 

3.  Display  Evaluation 

3.1  History  of  Vote/PeiTormancc  Evaluations 

The  following  comments  apply  to  evaluation  methods,  not 
to  the  particular  displays  or  display  concepts  involved. 
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Performance  based  studies-  In  the  1%0s,  United 
Kingdom  HUD  studies  were  performance-based  Naish 
measured  approach  tracking  performance  and  lateral  and 
glidcsiopc  errors  (ref  22)  One  conclusion  was  that 
director  symbols  and  slight  pitch  scale  compression 
improved  tracking  performance  The  shortcoming  of  the 
performance  measures  was  the  absence  of  measurements 
of  the  pilots’  ability  to  maintain  situational  awareness  in 
flight 

In  the  1970s,  Klopfstcin  developed  a  landing  symbology 
as  ar.  aid  to  flying  instrument  landing  system  (iiS) 
approaches  This  display  featured  a  synthetic  runway  (a 
runway  outline  whieh  appeared  over  the  Udi  runway)  and 
used  a  unique  angular  presentation  of  AOA  Pitots  who 
evaluated  this  display  reported  that  precise  airspeed 
control  and  tracking  performance  resulted  even  though  no 
airspeed  information  was  shown  on  the  HUD  (ref  23) 

The  conformal  runway  outline  has  been  used  m  mmt 
civilian  ILSHUDs  (refs  24  and  25). 

In  the  mid-19S0s,  the  U.S  Air  Force  studied  the  effect  of 
HUD  symbology  on  unusual  attitude  recoscry  and 
measured  a  variety  of  recovery  parameters  (ref  26)  The 
conclusions  supported  the  early  studies  and  recommended 
the  use  of  compressed  pitch  scale  and  a  recovery  cue 
This  study  also  indicated  that  air-mass  data  might  be 
beneficial  The  conclusions  lend  weight  to  the  need  for  an 
overall  objective,  performance-based  lest  methodology. 

In  spite  of  these  results,  there  has  been  reluctance  to  use 
the  compressed  pitch  scale,  the  synthetic  runway  outline, 
or  air-mass  data  in  operational  HUDs.  This  reluctance  has 
not  been  based  on  performance-based  evaluations,  but  on 
individual  pilot  opinions 

Opinion  based  decisions- The  AOA  bracket  and  the 
orientation  of  airspeed  and  altitude  scales  are  two  areas 
where  conflicting  opinion  has  created  divsimilar  formats 
to  display  the  same  information.  The  use  of  color  coding 
for  HDDs  is  another. 

At  one  point,  there  were  two  quasi-staiidards  for  color 
HDDs  developed  by  two  competing  transport  airplane 
manufacturers.  The  HDD  colors  differed  for  scales  and 
navigation  symbols.  On  review,  it  appeared  that  once  the 
decision  to  have  the  sky  color  be  blue,  the  warning  color 
be  red,  and  so  forth,  had  been  made,  only  a  limited 
number  of  choices  remained.  For  example,  if  the  sky  is 
blue,  the  pitch  scales  cannot  be  blue  also.  Each  company 
made  a  slightly  different  choice  for  various  scales 
resulting  In  non-uniform  colors. 

A  standardization  meeting  several  years  ago  seriously 
proposed  that  a  committee  take  an  equal  number  of 
choices  from  each  company’s  list  and  arrive  at  an 
**acccptablc  compromise.’’  The  alternative  was  to  choose 


between  the  two  companies  A  performance-based 
evaluation  was  not  discussed 

3.2  Subjective  Data 

Subjective  pilot  ratings  play  a  key  role  m  any  display 
evaluation  Historically,  pilot  ratings  have  been  patterned 
after  one  of  two  forms  the  traditional  Likert  difficulty 
scale  (ref  27)  or  the  Cooper-Harper  Pilot  Rating  (CHPR) 
(ref  28). 

Likert  rating  scales-  Traditional  rating  scales  ask  the 
pilot  to  rate  the  difficulty  making  choices  as  “very  easy,” 
“easy,”  “medium,”  “hard,”  or  “very  hard  "  A  derivative  of 
this  type  of  scale  is  the  task  load  index  (TLX)  rating  scale 
developed  by  NASA  (ref.  29)  Similar  ratings  were  used 
in  previous  HUD  simulations  (ref  26)  The  chief  advan¬ 
tage  for  a  Likert  scale  is  the  case  with  which  a  subject  can 
learn  them  It  can  also  be  useful  for  troubleshooting  an 
unacceptable  display 

One  disadvantage  of  such  scales  is  the  reluctance  of 
general  subjects  to  usc  extreme  values  and  the  reluctance 
of  pilot  subjects  to  use  “difficult”  ratings  unless  the 
display  IS  quite  difficult  to  fly.  As  a  result,  a  seven  point 
scale  frequently  becomes  a  three  point  scale. 

Cooper-Harper  pilot  ratings-  The  CHPR  scale  uses  a 
decision  tree  to  allow  the  pilot  to  “walkthrough”  a  series 
of  dichotomous  alternatives,  by  answering  questions,  such 
as  ”ls  It  {the  airplane]  coniioilablc?”,  “Is  adequate  perfor¬ 
mance  attainable  with  a  tolerable  workload?”;  and  “Is  it 
satisfactory  without  improvement'’”  Following  these 
dichotomies,  the  pilot  then  makes  a  choice  of  three  sub- 
altcrnativcs. 

The  main  advantage  of  this  approach  is  that  the  logic  tree 
involved  produces  consistent  results — particularly  with 
trained  evaluators  This  is  evident  in  the  area  of  aircraft 
handling  qualities  ratings. 

A  second  advantage  of  the  logic  tree  approach  is  apparent 
when  evaluations  arc  conducted  without  a  control  display 
or  control  symbology  In  this  case,  wc  don't  compare 
preferences,  but  determine  if  the  performance  objectives 
arc  met  and  what  degree  of  pilot  workload  is  required  to 
meet  them. 

The  major  difficulty  is  the  time  that  an  novice  evaluator 
must  spend  learning  the  logic  tree.  When  using  CHPRs 
with  untrained  evaluators,  quite  often  a  copy  of  the  logic 
diagram  is  provided  as  an  in-flight  aid  (ref.  18).  Scales 
based  on  CHPR-type  logic  trees  have  been  used  during 
low  altitude  navigation  targeting  infrared  for  night 
(LANTIRN)  evaluations  (ref  30).  A  similar  scale,  the 
Bedford  workload  scale,  was  used  tn  the  United  Kingdom 
for  HUD  evaluations  (personal  communication  with 
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J.  Hall,  Royal  Aircraft  Establishment,  Bedford,  England, 
1990,  and  ref  31). 

It  is  imperative  that  a  rating  be  taken  in  the  context  of  a 
specific  flight  segment  flown  by  a  typical  operational 
pilot.  Cooper  and  Harper  (ref.  28)  emphasized  this 
requirement,  but  it  applies  to  all  aircraft  control-display 
evaluations  as  well  When  using  a  task-oriented  evalu¬ 
ation,  the  evaluator  must  use  consistent  performance 
standards  The  standards  should  be  related  to  operational 
standards,  but  must  be  clearly  stated.  Table  3  shows 
examples  of  such  performance  standards 

3J  Display  Evaluation 

There  arc  two  aspects  of  flight  displays  that  must  be 
considered:  can  the  pilot  deterrrine  the  value  of  a  specific 
parameter,  such  as  airspeed?;  and  can  the  display  be  used 
to  control  that  variable?  These  two  questions  must  be 
answered  in  the  context  of  a  specific  task  scenario. 
Because  of  the  wide-spread  acceptance  of  the  CHPR  scale 
in  the  flight  test  community,  two  logic  trees  were  con¬ 
structed  to  rale  the  readability  and  the  controllability  of 
displays  (fig.s.  1  and  2).  An  earlier  version  of  those  figures 
(ref,  33)  was  used  by  the  U.S  Army  Center  for  Night 


Vision  and  Electro-Optics  for  Display  Flight  Assessment 
(ref.  34)  The  readability  rating  can  also  be  applied  to  the 
ease  of  overall  maintenance  of  situational  awareness  or 
attitude  awareness. 

These  display  ratings  follow  the  original  Cooper-Harper 
decision  tree  closely  The  difference  between  the  display 
flyability  rating  and  a  handling  qualities  CHPR  is  the 
requirement  that  the  evaluation  pilot  consider  aircraft 
control  using  the  display  for  information  This  is  essen¬ 
tially  a  CHPR  of  the  airplane  handling  qualities  in  series 
with  the  display  control  laws.  This  rating  for  a  given 
symbology  will  be  expected  to  vary  from  aircraft  to 
aircraft 

3.4  Additional  Questions 

In  addition  to  the  basic  rating  cards,  questions  should  be 
a5dkcd  addressing  specific  test  issues,  such  as  perceived 
problems  with  a  particular  display.  These  can  be  asked  at 
the  same  time  the  rating  card  is  completed  (following 
each  data  run)  or  during  a  debriefing  session.  The  final 
debriefing  questionnaire  should  also  ask  for  comparisons 
between  the  different  diNplays. 


Table  3.  Evaluatiem  task  performance  standards 


Desired  performance  standards 

Adequate  performance  standards 

Dynamic  maneuvers 

<2  see  to  acquire  new  attitude 
<5*  heading  and  roll  error  at  key 
points  during  maneuver. 

<3*  heading  error  on  recovery 
<100  ft  altitude  loss. 

<4  see  to  acquire  new  aliiiudc. 

<10*  heading  and  roll  error  at  key 
points  during  maneuver. 

<5*  heading  error  on  recovery. 

<200  ft  altitude  loss  No  PIO. 

Unusual  attitude  recoveries 

<1 .4  see  to  initial  airrcct  control 
input.  Initial  control  input  in 
accordance  with  published  instrument 
standards  (ref.  32).  No  control 
reversals.  No  overshoots  on  recovery 
to  wincs-lcvcl 

<1.8  see  to  initial  correct  control 
input.  Initial  control  input  In 
accordance  with  published  instrument 
standards  (ref.  32)  Single  control 
reversal  Single  overshoot  on  recovery 
to  wincs-lcvci. 

instrumcnl  approach 

Loc/GS  error  <0.5  dot. 

Airspeed  error  <2  knots  for  50%  of 
task.  No  overshoots  on  intercept. 

Go  around  at  DH  -f20/-4)  ft 

Loc/GS  error  <1  dot. 

Airspeed  error  <5  knoLs  for  entire 
task.  Single  overshoot  on  intercept 

Go  around  at  DH  -f  40/-0  ft 
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Figure  1.  Display  readability  rating  scale. 
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Figure  2.  Display  controllability  rating  scale. 


4.  Evaluation  Flight  Tasks 

AH  aircraft  have  many  common  mission  segments 
takeoff,  climb,  cruise,  descent,  terminal  area  maneu¬ 
vering,  approach  to  land,  hover,  and  landing  For  the  most 
part,  the  problems  during  these  common  mission  'cg- 
mcnls  arc  universal,  it  has  been  said  that  most  of  our 
problems  occur  in  the  last  1 5  miles  of  the  flight  (ref  35) 

All  mission  tasks  should  be  further  divided  to  separate 
visual  flight  from  instrumental  flight  Each  display  has  ns 
particular  set  of  problems 

When  evaluating  digital  flight  controls,  the  control  system 
may  be  acceptable  during  routine  mission  tasks,  but 
highly  unacceptable  during  aggressive  tracking  ta.sks  'this 
IS  described  as  a  handling  qualities  “cliff  ”  As  the  pitot 
tracks  more  and  more  aggressively,  the  handling  qualities 
deteriorate  quite  suddenly  and  sharply,  that  is,  falls  off  the 
cliff.  This  IS  often  more  pronounced  during  the  landing 
flare  or  aerial  refueling  tasks  (ref  36) 

Similarly,  digital  display  dynamics  can  result  tn  cliffs 
wnen  evaluated  during  aggressive  tracking  tasks  For 
example,  a  velocity  vector  symbol  may  be  well  behaved 
until  the  pilot  increases  his  gain  to  place  it  on  a  particular 
spot  on  the  runway  For  this  reason,  at  least  some  of  the 
experimental  tasks  should  require  aggressive  tracking  on 
the  part  of  the  subject  pilots. 

4,1  Evaluation  Task  Requirements 

Evaluation  tasks  should  be  appropriate  to  the  aircraft 
missions  Regardless  of  the  mission,  basic  instrument  ami 
visual  tasks  must  be  flown,  even  if  the  display  is  intended 
for  mission  specific  tasks  only. 

The  tasks  include  aggicssivc  pilot  tracking  to  test  the  cliff 
Low  level  terrain  following,  A/G  tracking,  landing  flare, 
and  unusual  attitude  recovery  are  examples  of  tasks 
requiring  aggressive  pilot  inputs  For  HDDs  and  HMDs, 
both  instrumental  and  visual  tasks  should  be  flow.). 

It  IS  also  essential  that  dynamic  maneuvering  against  real 
world  backgrounds  be  flown,  particularly  when  evaluating 
non-conformal  pitch  scaling  or  the  effect  of  clutter. 

Rights  against  a  real  world  background  should  be  flown 
both  day  and  night. 

There  must  be  some  performance  ba.Ms  with  which  to 
compare  different  displays.  Tracking  accuracy  is  often 
used  as  a  mca.surc  Unusual  attitude  recovery  uses 
reaction  time  to  the  first  correct  control  input  and  the 
number  of  control  reversals  during  the  recovery 


4.2  Evaluation  Tasks 

The  following  tasks  have  been  used  in  a  variety  of  studies 
and  arc  recommended  as  candidate  evaluation  tasks 

Unusual  attitude  recovery-  This  task  involves  a 
recovery  from  an  unusual  attitude  using  only  HUD/HMD 
symbology.  The  airplane  is  placed  in  an  unusual  attitude 
and  the  subject  pilot  is  directed  to  recover  to  a 
predetermined  heading  and  altitude. 

The  head-down  instruments  arc  covered  during  this  task 
and  view  of  the  real  world  cues  blocked  by  the  bluc/ambcr 
system  or  another  vision  restriction  device  During  the 
entry  into  the  unusual  attitude,  the  HUD  is  blanked 

Additional  unusual  altitudes  arc  introduced  during  other 
ta.sk$,  if  possible  For  example,  during  a  simulated  aii-to- 
air  tracking  task,  all  external  visual  cues  can  be  removed 
as  though  the  target  airplane  flew  into  a  cloud.  The  pilot 
has  to  recognize  the  situation  and  recover 

Dynamic  maneuvering  This  task  involves  aggressive 
instrument  flight  using  only  HUD/HMD  symbology.  The 
pilot  IS  asked  to  fly  a  senes  of  maneuvers  appropriate  to 
the  airplane.  Vertical  S  maneuvers  modified  to  include 
abruf^  changes  of  pitch  and  bank  arc  suitable  for  this  task. 
Instrument  acrobatics  (steep  turns,  barrel  rolls,  clover- 
leafs)  arc  also  used.  At  intervals,  the  subject  pilot  is 
distracted  with  a  task  requiring  head-down  viewing,  such 
as  reading  a  table  arranged  b)  rows  and  columns 
(personal  communication  with  J  Hall,  Royal  Aircraft 
Establishment,  Bedford,  England.  1990). 

Aimpoint  tracking-  Air-to-ground  weapons  delivery  is  a 
highly  suitable  experimental  task  for  HUDs  and  HMDs.  It 
requires  aggressive  tracking  on  the  pan  of  the  subject 
pilot  For  transports,  a  related  task  is  a  visual  approach  to 
landing  requiring  the  pilot  to  maintain  a  specific  aimpoint 
With  the  flight  path  marker. 

This  task  helps  to  identify  any  problems  associated  with  a 
non-conformal  display. 

Instrument  approach- This  task  involves  an  approach  to 
a  landing  or  to  a  missed  approach.  Approximately  half  of 
the  approaches  arc  to  a  landing  and  half  to  a  mis.scd 
aj^roach  Both  precision  and  non-prccision  approaches 
arc  flown 

Vbual  approach-  Ihis  task  involves  a  vksual  approach  to 
a  landing.  Some  approaches  arc  flown  at  night  and  both 
siraight-in  and  circling  approaches  arc  flown. 

System  failures-  During  any  of  the  tasks,  it  is  important 
to  consider  the  effect  of  system  or  sensor  failures.  ILS 
approaches  should  induce  single  axis  failures  (such  as 
glidcsiopc  (GS)  failure)  and  deternine  if  the  pilot  can 
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ccognizc  this  event  and  maintain  suitable  performance 
following  the  failure 

Choice  of  Pilots 

One  fundamental  question  is*  Should  test  pilots  or 
operational  pilots  be  used  as  evaluators? 

Arguments  favoring  operational  pilots  include  having 
pilots  with  recent  mission  experience.  It  is  also  possible  to 
obtain  a  range  of  experience  levels,  from  recent  pilot 
training  graduates  to  experienced  pilots 

One  problem  with  using  operational  pilots  is  that  each 
pilot  is  often  overtrained  on  a  particular  display  and  may 
be  predisposed  to  that  display~F*16  pilots  prefer  F*i6 
symbology,  whereas  F-1S  pilots  prefer  F-18  symbology. 
Ideally,  operational  pilots  with  no  symbology  background 
should  be  used.  Unfortunately,  this  is  not  possible.  To 
avoid  this  problem,  the  experimenter  must  ensure  that  no 
particular  symbology  is  overrepresented  and  that 
subjective  data  arc  used  with  cate. 

Another  problem  is  the  need  to  train  operational  pitots, 
both  in  how  to  fly  with  non-standard  displays  or  tech¬ 
niques  and  in  how  to  use  rating  scales.  It  is  imperative  that 
adequate  familianzation  and  instructions  are  provided. 
This  is  most  apparent  with  scales  similar  to  the  CHPR. 

The  training  can  amount  to  two  or  three  practice  sorties 
per  pilot  compared  with  one  for  a  trained  evaluator. 

Arguments  favoring  test  pilots  include  having  trained 
evaluators.  Properly  trained  test  pilots  arc  used  to  rate 
airplane  handling  and  should  be  familiar  with  rating  scales 
such  as  the  Cooper-Harper  type  of  walk-through  ratings. 
Test  pilots  are  also  skilled  ai  communicating  with 
engineers  and  can  provide  insight  into  display  or  control 
taw  problems. 

Test  pilots  are  experienced  pilots,  although,  perhaps,  not 
with  recent  mission  experience.  They  usually  hive  a 
broad  range  of  experience  in  different  airplanes  and  with 
different  displays  This  allows  them  to  be  able  to  adapt 
their  individual  control  strategics  to  the  display,  such  as 
using  the  pitch  symbol  versus  velocity  vector  symbol  for 
aircraft  control. 

The  test  pilot  must  remain  objective.  Special  care  must  be 
taken  if  a  test  pilot  has  had  a  major  role  in  designing  the 
symbology.  In  this  case,  it  would  he  best  for  the  test  pilot 
to  be  disqualified  from  the  final  approval  portion  of  the 
tests. 

The  need  to  conduct  practice  sorties  for  untrained 
evaluators  can  quickly  use  up  the  available  sorties  in  a 
program.  For  example,  if  24  sorties  arc  available,  using 
two  test  pilots  will  allow  for  22  data  sorties.  If  six 


(^rational  pilots  arc  used  instead,  12  to  18  practice 
sorties  may  be  required  allowing  only  six  to  12  data 
flights. 

If  the  display  is  novel  or  controversial,  it  may  be 
necessary  to  use  pilots  with  varying  experience  as  a  final 
check,  although  this  will  not  normally  be  ncccs.sary. 

5,  Display  Issues 

5.1  Symbology 

There  are  a  number  of  symbology  issues  worth 
examining.  However,  space  will  limit  the  discu.ssion  to 
two  current  HUD  issues. 

One-to-one  versus  compressed  sealing-  For  some  time. 
It  has  been  axiomatic  in  HUD  designs  that  the  display 
should  be  in  one-to-one  scaling  with  the  outside  world. 
While  there  is  no  doubt  that  stch  scaling  is  a  considerable 
benefit  to  the  pilot,  there  is  also  a  growing  amount  of 
research  indicating  a  benefit  for  compressed  pitch  scaling. 

The  main  advantage  of  1:1  scaling  is  that  the  pilot  can 
immediately  visualize  his  airaaft’s  trajectory  if  the  HUD 
shows  an  inenial  velocity  vector.  One-to-one  scaling  also 
allows  for  very  precise  determination  of  aircraft  pitch 
attitude  and  immediate  \  isualization  of  the  aircraft’s  angle 
of  attack  (AOA)  with  an  air  mass  system. 

Dunng  ground-referenced  maneuvers — A/C  weapons 
delivery,  low  level  navigation,  approaches  to  landing,  or 
obstruction  critical  takeoffs — visualization  of  the  aircrafl 
trajectory  is  critical  Using  a  scaled  longitudinal  accelera¬ 
tion  to  Visualize  aircraft  trajectory,  the  pilot  can  determine 
the  steady-state  climb  cap7.bil:ty  of  his  aircraft.  Such  a 
potential  flight  path  can  be  beneficial  during  engine-oul 
climbs,  fo.'  example. 

At  the  same  lime,  early  HUD  research  in  the  United 
Kingdom  indicated  that  a  pilot  could  fly  a  trajectory  much 
more  precisely  if  the  pitch  scaling  were  reduced  to  1.5:1 
or  2:1.  This  was  m  spile  of  not  being  able  to  detect 
.smaller  deviations  as  with  1  1  scaling  (ref.  22). 

Also,  recent  investigations  into  spatial  disonentatioii 
indicates  that  compressed  pilch  scaling  may  help  mini¬ 
mize  the  tendency  to  suffer  spatial  disorientation  and  may 
aid  the  pilot  during  unusual  attitude  recoveries  (ref.  26). 
This  same  study  suggested  that  a  1:1  HUD  near  the 
horizon  combined  with  compressed  scaling  away  from  the 
horizon  might  be  an  acceptable  compromise.  Both  con¬ 
tinuously  varying  compression  and  a  step  change  have 
been  suggested.  A  step  change  is  presently  implemented 
in  the  F-16  HUD  (personal  communication  with 
D.  Howlings,  GEC  Avionics,  Aug.  1991). 


11 


With  these  observations,  there  is  a  definite  need  to 
experiment  to  determine  the  effect  of  various  MUD  pitch 
scalings  The  experiments  must  be  performed  in  flight 
simulation  and  later  in  an  airplane  to  validate  or  reject  the 
simulator  results.  The  experiment  should  explore  the 
effect  of  variations  in  scaling,  including  both  gradual  and 
step  changes.  The  effect  of  pitch  scale  compression 
(including  variable  gearing  and  step  changes)  should  he 
evaluated  during  all  ground-reference  maneuvers 

Air  mass  \crsus  inertial  data-  Recent  research  indicates 
that  inertial  quality  attitude  data  are  essential  for  HUDs 
Designers  have  implictlly  assumed  that  this  requires  the 
use  of  inertial  velocity  vectors  as  well 

The  advantage  of  an  incrtui  velocity  vector  is  the  direct 
display  of  the  aircraffs  trajectory  against  the  real  world 
For  example,  v/hen  coupled  with  1.1  scaling,  the  pilot  can 
quickly  determine  exactly  where  the  airplane  is  going, 
particularly  during  the  final  .ipproach  to  landing. 

At  the  same  time,  using  an  air  mass  velocity  vector 
presents  direct  viewing  of  the  aircraft’s  AOA.  Air  mass 
velocity  vector,  Klopfstcin  allowed  pilots  to  fly  more 
precise  final  approaches  in  terms  of  airspeed  control  and 
ILS  tracking  accuracy  (ref.  19) 

There  is  no  question  that  pilots  need  to  be  aware  of  the 
aircraft  performance  in  terms  of  the  air  mass.  The  issue  is 
whether  or  not  the  benefits  of  displaying  an  air  mass 
velocity  vector  is  more  important  than  the  benefit  of 
having  a  velocity  vector  conformal  to  the  real  world.  If 
compressed  pitch  scaling  becomes  commonplace,  the 
effect  of  a  conformal  /clocity  vector  may  be  less  apparent 
and  may  well  influence  the  result.  The  issue  must  be 
evaluated  with  performance  data  and  will  certainly 
depend  upon  specific  maneuvers  and  tasks. 

S.2  Display  Dynamics 

In  modern  aircraft,  the  pilot  obtains  much  of  his  flight 
information  through  the  cockpit  displays.  It  is  not  easy  to 
separate  display  control  laws  from  the  aircraft  handling 
qualities.  The  display  dynamics,  the  scat-of-thc-panis  feel, 
and,  during  visual  meteorological  conditions  (VMC),  the 
view  of  the  real  world  all  form  part  of  the  feedback  Uxip. 

All  of  these  feedback  loops  must  he  considered  when 
performing  evaluations.  Traditional  handling  qualities 
evaluations  only  consider  the  aircraft  dynamics  with  the 
motion  and  external  vision  feedback  loops.  Since  these 
loops  arc  essentially  instantaneous,  display  dynamics  do 
not  affect  the  results. 

Traditional  instrument  handling  qu.s!iiics  evaluations  used 
conventional  instruments  and  benign  instrument  tasks 
Because  of  the  inherent  damping  in  conventional  instru¬ 


ments,  and  since  typical  instrument  tasks  arc  not  very 
aggressive  in  nature,  the  instrument  display  dynamics  do 
not  interact  with  handling  qualities  demonstrations 

However,  modem  aircraft  arc  t>eing  flown  in  aggressive 
mancuvcis  by  reference  to  their  displa)s.  Pilots  are 
dependent  on  the  on-board  sensors  and  associated  dis¬ 
plays.  Even  in  VMC.  the  presence  of  an  HUD  or  HMD 
ensures  lual  the  display  dynamics  cannot  be  ignored  by 
the  test  pilot  as  they  were  in  traditional  VMC  C''aluations 
The  display  has  become  an  integral  part  of  the  aircraft  and 
the  display  dynamics  pari  of  the  overall  control  laws 
governing  handling  qualities 

The  computer  cycle  time,  or  frame  time,  is  an  area  of 
particular  concern  Early  as  ionics  were  electrical  analogs 
of  motion  equations.  An.ilog  computers  have  the  advan¬ 
tage  of  being  much  faster  than  digital  computers  and  can 
process  multiple  functions  in  parallel  Digital  computers, 
on  the  other  hand,  procos  multiple  channels  in  scries.  The 
digital  display  compuicr  has  a  defined  cycle  between 
20  and  lOU  msec. 

Data  sampling  will  also  adversely  affect  display 
dynamics.  For  example,  if  a  given  sensor  input  signal  is 
sampled  every  50  msec  and  this  value  is  used  to  calculate 
the  output  signal  that  appears  50  msec  later  at  the  end  of 
the  cycle,  two  effects  happen.  Brst,  the  output  is  delayed 
50  msec;  second,  the  input  and  output  are  sampled  every 
50  msec.  The  waveforms  of  the  signals  arc  changed  to 
reflect  a  senes  of  step  functions,  not  the  smooth  curves  we 
expect  The  sampling  introduces  high  frequency  ‘*noise,’* 
which  contaminates  the  input  signal  in  addition  to 
eliminating  signal  information  at  frequencies  higher  than 
the  sampling  frequency. 

6.  Conclusion 

The  rapid  evolution  of  graphics  technology  allows  greater 
flexibility  m  aircraft  displays,  but  display  evaluation 
techniques  have  not  kept  pace  Display  evaluation  in  the 
past  has  been,  to  a  large  extent,  based  on  subjective 
opinion  and  not  on  the  actual  aircrafl/pilot  performance. 
Pilot  opinion,  while  valuable,  must  be  tempered  with 
performance  measurement 

Modern  digital  displays  interact  strongly  with  aircraft 
dynamics  and  cannot  be  easily  separated  from  the  aircraft 
handling  qualitic.s.  Many  of  the  issues  in  fly-by-wire  flight 
control  systems  arc  similar  to  flight  displays  issues.  While 
technology  allows  the  designer  great  opportunities  to 
tailor  the  display  to  the  mission,  it  also  allows  for  greater 
f^^rtuniiics  for  creating  an  unworkable  system. 

Display  evaluation  is  not  an  simple  task  It  requires  as 
much  attention  as  any  other  flight  critical  system  Some  of 
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the  problems  high  performance  aircraft  have  exhibited  tn 
terms  of  lack  of  situational  awareness  or  spatial  dtsonen* 
tation  have  had  their  origins  tn  poor  display  design.  The 
community  is  concerned  that  informal  discussions  still 
form  the  basis  for  many  symbology  standards. 

Future  display  evaluation  methodology  is  not  *‘ca.st  in 
concrete*’  and  is  still  in  the  developmental  stage.  The  test 
and  evaluation  community  should  be  challenged  to 
participate  in  the  discussions  that  arc  sure  to  follow, 
including  the  choice  to  use  operational  or  test  pilots,  the 
choice  of  performance  metrics  and  criteria,  the  choice  of 
subjective  rating  scales,  the  need  for  standardization,  as 
well  as  the  many  display  issues  themselves. 

Display  rating  techniques  analogous  to  handling-quality 
ratings  presented  in  figures  1  and  2  were  developed  and 
arc  recommended  for  future  evaluations.  These  display 
rating  techniques  were  constructed  to  rate  the  readability, 
interpretability,  and  controllability  of  display  symbology. 
An  earlier  version  of  these  rating  techniques  was  used  to 
perform  Integrated  Helmet  Display  Flight  Asses.sment  by 
the  U.S.  Army  Communication  Electronics  Command, 
Airborne  Electronics  Research  Detachment  (ref.  34) 

It  is  important  to  recognize  that  while  standardization  is  a 
desirable  goal,  there  arc  good  and  valid  reasons  to  deviate 
from  a  standard:  new  missions,  new  technology,  new 
graphics  processors,  and  different  aircraft  We  do  not  wish 
to  constrain  the  design  of  new  displays,  but,  rather,  give 
the  designer  and  the  evaluator  the  display  tools  to  allow 
them  to  make  reasonable  choices  knowing  the  effect  of 
their  choices  on  the  resulting  performance 
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Abbreviations 

HDD 

Head-down  display 

A/G 

Air  to  ground 

HMD 

Helmet-mounted  display 

AOA 

Angle  of  attack 

HUD 

Head-up  display 

CHPR 

Cooper-Harper  Pilot  Rating 

Its 

Instrument  landing  system 

CRT 

Cathode  ray  tube 

IMC 

Instrument  meteorological  conditions 

DH 

Decision  height 

lANTIRN 

Low  Altitude  Navigation  and  Targeting 

DME 

Distance  measuring  equipment 

Infrared  for  Night 

FUR 

Forward-looking  infrared 

no 

Pilot-induced  oscillation 

FOV 

Field  of  view 

PNVS 

Pilot  Night  Vision  Sensor 

GS 

Glideslope 

TLX 

Task  Load  Index 

VMC 

Visual  meteorolo^cal  conditions 

15 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

0MB  No  0704’018e 

PuMc  r«porwt9  biaotn  tor  vm  cofcfton  cT  mtormnon  »  MtinaiM  to  avorogo  i  new  pot  iMponoo  MtuOng  ti*  am*  tot  rovtowng  n•l^xMn•  Morcnitv;  onotmo  oau  aourcM 
gothotng  and  miww»ng  tf»  doU  noodoo  and  comptoano  and  favta<«w>g  tho  eottocton  ot  ifitetfnaMn  Sand  catwrtanis  rogardo^g  tM  bgrdon  aabmait  or  any  othat  aapaet  ol  Vm 
coilaebon  o(  Mermabon  ndwdatg  auggatbena  for  radwcvtg  tfna  bi,adan  lo  ^ashtngun  Haadduartara  Samcaa  Ooactorata  lor  i.^mabot' Opa^akono  and  Rapens  12IS  Jatfaraon 
DanoHgharay  Seda  1204  Artm^  VA22202-4302  and  to  lha  OKca  o' Managantam  and  Sgogai  Paparwork  RadwebonP^aci  (0704-01$^  Washington  0C20S03 

1.  AGENCY  USE  ONLY  ^Uaira  8Im*; 

4.  TITLE  AND  SUBTITLE 

Test  Techniques  for  Evaluating  Flight  Displays 

9  FUNDING  NUMBERS 

505-64-36 

S  AUTHOR(S) 

Loran  A.  Haworth  and  Richard  L.  Newman 

7  PERFORMING  ORGANIZATION  NAME(S)  AND  AOORESS(£S) 

AeroRightdynamics  Directorate.  U.S.  Army  Aviation  and  Troop 

Command,  Ames  Research  Center,  Moffett  Field,  CA  94035-1099  and 

Ames  Research  Center,  Moffett  Field,  CA  94035-1000 

9  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

A-92139 

9  SPONSOniNQ/MONITORINQ  AGENCY  NAME(S|  AND  AOOR£SS(ES) 

National  Aeronautics  and  Space  Administration, 

Washington,  DC  20546-0001  and 

U.S.  Army  Aviation  and  Troop  Command,  St.  Louis,  MO  63120-1798 

10.  SPONSORtNG/MONITORINO 

AGENCY  REPORT  NUMBER 

NASATM-103947 
USAATCOM  TR.92-A-006 

11.  SUPPLEMENTARY  NOTES 

Point  of  Contact:  Loran  A.  Haworth,  Ames  Research  Ontcr,  MS  243-4,  Moffett  Field,  CA  9403S-1000 
(415)604-6944 

124.  OISTRIBUTION/AVAILABILITY  STATEMENT 

Unclassified  —  Unlimited 

Subject  Category  05 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (M§Mlmum  TOO  wordMj 

The  rapid  development  of  graphics  technology  allows  for  greater  flexibility  in  aircraft  displays,  but 
display  evaluation  techniques  have  not  kept  pace.  Historically,  display  evaluation  has  been  based  on  subjec¬ 
tive  opinion  and  not  on  the  actual  aiicraft/pilot  performance.  Existing  electronic  display  specifications  and 
evaluation  techniques  are  reviewed.  A  display  rating  technique  analogous  to  handling  qualities  ratings  has 
been  developed  and  is  rcccmmcndcd  for  future  evaluations.  The  choice  of  evaluation  pilots  is  also  discussed 
and  the  use  of  a  limited  number  of  trained  evaluators  is  recommended  over  the  use  of  a  larger  number  of 
operational  pilots. 

It.  SUBJECT  TERM, 

Flight  testing.  Flight  symbology.  Helmet-mounted  display,  HMD,  Head-up  display, 
HUD,  Flight  displays.  Electronic  display  standards.  Display  evaluation.  Display 
readability  rating  scale.  Display  controllability  rating  scale 

IS.  NUMBER  OF  PAGES 

18 

19  PRICE  CODE 

A02 

17.  SECURITY  CLASSIFICATION 
OF  REFORT 

Unclassified 

HCB 

20.  LIMITATION  OF  ABSTRACT 

MSN  7S40^i  >2ao>S900  Standard  Form  2M  (Rav  2-89) 


m-lt 


